Method and system for recording and synchronizing audio and video signals and audio and video recording and synchronization system

ABSTRACT

There is set forth a method and a system for recording and synchronizing audio and video signals. The audio signal and the video signal are stored together with time stamps from a respective associated system clock. The invention relates to an adaptation of the duration of the recorded audio sequence to the duration of an associated video sequence in order to level out differences in synchronization of the two system clocks. Alignment of the two system clocks is also introduced, which is based on a data transfer which has variable waiting times for the access to a transmission channel. This thus permits clock alignment with means as are available for example on a smartphone.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is the National Stage entry under 35 U.S.C. § 371 ofInternational Application No. PCT/EP2018/076489 filed Sep. 28, 2018,published as Publication No. WO 2019/072598 on Apr. 18, 2019, whichclaims benefit of foreign priority of German Patent Application No. 102017 123 319.9, filed on Oct. 9, 2017, the entireties of which areherein incorporated by reference.

FIELD OF DISCLOSURE

The present invention concerns a method and system for recording andsynchronizing audio and video signals.

BACKGROUND

Mobile devices like smartphones are frequently equipped with a powerfulcamera. A zoom function makes it possible to film even events at arelatively great distance from the camera. That can involve for examplea performance on the stage in a school auditorium, in which the user ofthe mobile device (camera unit) is sitting in the audience and makes avideo recording of what is happening on the stage. An audio recordingusing a microphone included in the camera unit often givesunsatisfactory results as much unwanted ambient noise from the directsurroundings of the camera unit is recorded while the desired audiosignal—for example the words spoken on the stage—is recorded too quietlyand is lost in the ambient noise.

An external microphone can be used as a remedy for that. That can be forexample a clip-on microphone which a performer on the stage has fastenedto his clothing and which records the audio signal which is spoken orsung by the performer or an overall audio signal from the immediatevicinity thereof. If an external microphone is connected to the cameraunit by way of a cable the audio signal from the external microphone canbe stored directly at the same time as and thus synchronously jointlywith the recorded video signal. In many situations of use—for example inthe auditorium—wired transmission of the audio signal however is notpracticable.

There is a wish to be able to receive the wirelessly emitted audiosignal without additional hardware with a normal camera unit like forexample a smartphone. Individualized wireless data transfer protocolsare available for such mobile end-user devices. Examples that may bementioned here are WLAN, Bluetooth, DECT, LTE or Wi-Fi, in which a datastream is respectively provided for a given end device and transmittedsubdivided into individual data packets to the receiving user device. Inthe case of those data transfer protocols however an unpredictable timedelay is tolerated in the transfer of each individual data packet as theuse of a transmission channel in accordance with the protocol iscoordinated as between a plurality of transmitting devices so thatvariable waiting times are involved for access to the transmissionchannel. In addition the transfer is normally bidirectional, in whichrespect a receiving device acknowledges successful reception and in theevent of faulty transfer a data packet is transmitted afresh, which inaddition results in unpredictable delay. Overall a delay of severalseconds can occur. An audio signal received in that way is time-shiftedby an unknown amount in relation to the video signal recorded by thecamera unit, to such an extent that the time shift is perceived as beingdisturbing upon later consideration of the video/audio signal. Withoutadditional synchronization measures it is not possible for the audiosignal to be subsequently shifted to the correct place with respect tothe video signal.

As a further alternative to wireless transfer storage of the audiosignal in the external microphone unit can also be considered. The audiosignal from the external microphone unit is then subsequently broughttogether with the video signal from the camera unit. In this case alsothe problem that arises is that of subsequently bringing the audiosignal to the correct place on the time axis with respect to the videosignal.

DE 10 2010 038 838 A1 discloses a wireless microphone system in which amicrophone unit records an audio signal and stores it jointly with “timestamps” and wherein a camera unit stores corresponding time stampsjointly with the video signal and wherein the camera unit emitssynchronization signals, on the basis of which the microphone unitproduces time stamps matching the video signal.

WO 2016/162560 A1 discloses a method of synchronizing audio and videosignals by means of optical synchronization signals.

On the German patent application from which priority is claimed theGerman Patent and Trade Mark Office searched the following documents: US2015/0 104 151 A1, US 2016/0 269 136 A1, JP 2007-81 686 A andSCHMALENSTROEER, JOERG [et al:]: A gossiping approach to sampling clock,synchronisation in wireless acoustic sensor networks. In: InternationalConference on Acoustic, Speech and Signal Processing (ICASSP), Florence,Italy; IEEE Transactions, 14 Jul. 2014, pages 7575-7579.

SUMMARY OF INVENTION

An object of the present invention is to provide a method and system forrecording and synchronizing audio/video signals, which allow subsequentsynchronization of audio and video signals which belong together.

The object is attained by a microphone unit as set forth in claim 1, asystem of recording and synchronizing audio/video signals as set forthin claim 2 and a method of recording and synchronizing audio/videosignals as set forth in claim 3.

Thus there is provided a microphone unit having a microphone forrecording audio signals, a transmitter/receiver for wirelessbidirectional communication with a video unit which has a first systemclock with a first time base, a second system clock with a second timebase, and a memory for digital storage of an audio signal recorded withthe microphone and time synchronization information. The microphone unitis adapted to repeatedly store in the memory jointly with an audiosignal recorded by the microphone audio time stamps which specify therespective times of recording the audio signal measured with the secondsystem clock. The transmitter/receiver is adapted to communicate withthe video unit by way of a data transfer protocol which is provided fora coordinated use of a used transmission channel by a plurality oftransmitting and receiving devices and which has measures for timecoordination of the access of different devices to the transmissionchannel so that variable waiting times can occur for access to thetransmission channel. The microphone unit is adapted by way of thetransmitter/receiver to carry out an alignment between the first systemclock and the second system clock, wherein the alignment result includesa time value of the first system clock and an associated time value ofthe second system clock. The alignment is effected by the video unit orthe microphone unit as the initiating unit firstly reading out a currenttransmission time value from its system clock and temporarily storing itand immediately initiating a first transfer process to the other of thetwo units. The respective other unit after conclusion of the firsttransfer process immediately reads out a current synchronization timevalue from its system clock and transfers said synchronization timevalue in a second transfer process to the initiating unit. Theinitiating unit upon reception of the second transfer immediately readsa current reception time value from its system clock and from thetransmission time value and the reception time value calculates a timevalue belonging to the synchronization time value.

According to a further aspect of the invention there is provided anaudio/video recording and synchronization system comprising a microphoneunit and a video unit. The video unit includes a camera unit forrecording video signals, a transmitter/receiver for wirelessbidirectional communication with the microphone unit, a first systemclock with a first time base, and a memory for digital storage of avideo signal recorded with the camera unit and time synchronizationinformation. The video unit is adapted to repeatedly store in the memorytogether with a video signal recorded by the camera unit video timestamps which specify the respective times of recording of the videosignal measured with the first system clock. The video unit is adaptedby way of the transmitter/receiver to carry out an alignment between thefirst system clock and the second system clock.

According to a further aspect of the invention there is provided amethod of recording and synchronizing audio and video signals comprisingthe steps: recording a video signal by means of a video unit and storingthe video signal together with video time stamps which specify therespective times of recording of the video signal measured with a firstsystem clock, recording an audio signal by means of a microphone unitand storing the audio signal together with audio time stamps whichspecify the respective times of recording of the audio signal measuredwith a second system clock, carrying out an alignment between the firstsystem clock and the second system clock, wherein the alignment resultcontains a time value of the first system clock and an associated timevalue of the second system clock, and bringing together the audio signaland the video signal, wherein time synchronization is effected on thebasis of the video time stamps, the audio time stamps and the alignmentresult.

According to a further aspect of the invention alignments are carriedout repeatedly at different times between the first system clock and thesecond system clock and alignment results are stored.

According to a further aspect of the invention the method furtherincludes the step of again sampling the recorded audio signal with analtered sampling frequency to adapt the length of the audio signal tothe length of the video signal.

According to a further aspect of the invention the method additionallyincludes the steps: reading out the starting time of a video sequence onthe basis of the video time stamps, reading out the end time of a videosequence on the basis of the video time stamps, calculating an audiostart time in the audio recording which belongs to the starting time ofthe video sequence on the basis of the audio time stamps and alignmentresults, calculating an audio end time in the audio recording whichbelongs to the end time of the video sequence on the basis of the audiotime stamps and alignment results, re-scaling the audio sequence betweenthe audio start time and the audio end time so that the duration of therescaled audio sequence nominally coincides with the duration of thevideo sequence, and bringing the video sequence together with therescaled audio sequence.

According to a further aspect of the invention the alignment between thefirst system clock and the second system clock is effected by a pingpongsynchronization. A data packet is transmitted from the video unit to theaudio unit and the packet is then transmitted from the audio unit to thevideo unit in order to carry out a comparison of a system clock of thevideo unit and a system clock of the audio unit to ascertain a shiftbetween the system clock of the video unit and the microphone unit.

According to a further aspect of the invention the alignment between thefirst system clock and the second system clock is effected by way of abidirectional wireless connection between the video unit and themicrophone unit. In that case a data transfer protocol is used, which isprovided for a coordinated use of a used transmission channel by aplurality of transmitting and receiving devices and which has measuresfor time coordination of the access of different devices to thetransmission channel so that variable waiting times can occur for accessto the transmission channel. The alignment is effected by the video unitor the microphone unit as the initiating unit firstly reading out acurrent transmission time value from its system clock and temporarilystoring it and immediately initiating a first transfer process to theother of the two units, and wherein the respective other unit afterconclusion of the first transfer process immediately reads at a currentsynchronization time value from its system clock and transfers saidsynchronization time value in a second transfer process to theinitiating unit. The initiating unit upon reception of the secondtransfer immediately reads out current reception time value from itssystem clock and from the transmission time value and a reception timevalue calculates a time value belonging to the synchronization timevalue.

According to a further aspect of the invention a first alignment betweenthe first system clock and the second system clock is initiated at thebeginning of a video recording and a second alignment is initiated withthe end of the video recording.

Further configurations of the invention are subject-matter of theappendant claims.

Advantages and embodiments by way of example of invention are describedin greater detail hereinafter with reference to the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows a diagrammatic view of a system for recording andsynchronizing audio and video signals according to a first embodiment,

FIG. 1B shows a diagrammatic view of an external device in whichoptionally processing of video and audio data can be carried out,

FIG. 2 shows a diagrammatic view of system clocks of the video unit andthe audio unit,

FIG. 3 shows a further diagrammatic view of the system clocks of thevideo unit and the audio unit in the case of system clocks running atdiffering speeds,

FIG. 4 shows a diagrammatic view of processing of the audio recordingrecorded in accordance with FIG. 3 ,

FIG. 5 shows a diagrammatic view of a corrected audio data file andvideo data file according to an aspect of the present invention,

FIG. 6 shows a diagrammatic view of the first system clock and thesecond system clock in relation to time in accordance with an aspect ofthe present invention,

FIG. 7A shows a diagrammatic view of a synchronization between a videounit and an audio unit,

FIG. 7B shows a diagrammatic view of a synchronization between a videounit and an audio unit according to an aspect of the invention,

FIG. 8 shows a diagrammatic view of a synchronization measurementaccording to an aspect of the present invention, and

FIG. 9 shows a diagrammatic view of the offset of the system clocks ofthe video unit and the microphone unit, that is ascertained from asynchronization measurement.

DETAILED DESCRIPTION

FIG. 1 shows a situation of use of a microphone unit and a video unitaccording to a first embodiment of the invention. A performer 102 is onthe stage. A video recording of what is happening on the stage isproduced with a camera unit 10 from an auditorium. A microphone unit 200is to be found on the stage 101 at the performer 102. The microphoneunit can be for example a clip-on microphone, a hand-held microphone ora body unit connected to a microphone by way of a cable. Optionally themicrophone unit 200 can also be positioned separately from theperformer, for example on a microphone stand on the stage 101.

The video unit 100 in FIG. 1 has a transmitter/receiver 110, a firstsystem clock 120 with a first time base, a camera unit 130 for recordinga video signal and optionally a first synchronization unit 140. Thevideo unit 100 also has a memory 150 in which inter alia a recordedvideo signal can be stored. The video unit 100 is in particular a normalmobile end-user device like for example a smartphone, which allowsdigital recording of a video signal and which is provided for wirelessbidirectional transfer of digital data. For use in the context of theinvention suitable software can be installed on the video unit.

According to the first embodiment the microphone unit 200 has atransmitter/receiver 210, a second system clock 220 with a second timebase, a microphone 230 for recording an audio signal and a secondsynchronization unit 240. Recording of the audio signal of themicrophone unit is effected in digital form by sampling the signalrecorded by the microphone 230, using a sample rate which is produced inthe microphone unit 200.

The microphone unit 200 has a memory 250 in which inter alia a recordedaudio signal can be stored.

The transmitter/receiver 110 of the video unit 100 can communicatewirelessly with the transmitter/receiver 210 of the microphone unit 200.Detection of synchronization can be effected by the firstsynchronization unit 140 or by the second synchronization unit 240.

On the basis of the first system clock 120 the video unit 100 canrepeatedly store time information (video time stamps) together with arecorded video signal, the time information specifying the respectivetimes of recording of the video signal—measured with the first systemclock 120. Correspondingly, on the basis of the second system clock 220,the microphone unit 200 can repeatedly store time information (audiotime stamps) together with an audio signal recorded by the microphoneunit, which time information specifies the respective times of recordingof the audio signal—measured with the second system clock 220.

A first problem arises because the time shift between the first systemclock 120 and the second system clock 220 is normally unknown so thatthe time information recorded on the two devices is alone not sufficientto subsequently permit a correct association in respect of time betweenthe video signal and the audio signal. According to an aspect of thepresent invention therefore an alignment is carried out between the twosystem clocks 120 and 220, wherein the alignment is implemented by aparticular use of a data protocol which is available on an end-userdevice and which is actually not intended to ensure a predictable timeshift in the transmission. A more precise description of that use ofsuch a data protocol is set forth hereinafter with reference to FIGS. 7Athrough 9 .

Alignment is effected by simultaneously ascertaining the time value ofthe first system clock 120 and the second system clock 220. The twosimultaneously ascertained time values can then be stored jointly.Storage of the two associated time values can be effected in the memory250 of the microphone unit 200 and/or in the memory 150 of the videounit 100. By virtue of the access to that stored information it is laterpossible to carry out a correct time association between the videosignal recorded with video time stamps on the basis of the first systemclock 120 and the audio signal recorded with audio time stamps on thebasis of the second system clock 220. It is not necessary for thatpurpose to adjust one of the system clocks 120 and 220, but it is onlythe information in respect of the time values of the two system clocksthat is ascertained at a common time and stored.

Differences in Speed of the System Clocks:

A second problem arises out of the fact that the two system clocksnormally do not run at exactly the same speed. This can have the resultthat a video sequence which is recorded by the video unit 100 and whichbegins at a real start time and ends at a real end time appears longeror shorter in the measurement in accordance with the first system clock120, than the period which has actually elapsed. The same applies to thesecond system clock 220 so that the audio sequence which is recorded bythe microphone unit 200 and which begins at the real start time and endsat the real end time appears longer or shorter in measurement inaccordance with the second system clock 220 than the period which hasreally elapsed. Independently of the really elapsed period thereforewith system clocks 120 and 220 running at different speeds, the videosequence in measurement with the first system clock 120 is of adifferent duration from the associated audio sequence in the measurementwith the second system clock 220. In order to counter that problemalignment between the two system clocks is effected according to theinvention a plurality of times, that is to say at different times, andthe result is stored a plurality of times, as described above.Optionally a first alignment operation can be provided prior to or withthe beginning of a video recording and a second alignment operation canbe carried out with or after the conclusion of the video recording.Alternatively a alignment operation can be carried out before thebeginning of an event and a second alignment operation can be carriedout after the conclusion of an event. An alignment test is also usuallyplanned, in which case a result is produced only when a wireless orwired connection exists between the video unit 100 and the microphoneunit 200, which allows alignment.

On the basis of stored alignment results which were ascertained atdifferent times the unequal speed as between the first and the secondsystem clocks can be measured and that information can be processed whenbringing the video and the audio signals together. If for example avideo signal in the measurement with the first system clock 120 involvesa duration of 1000 seconds and an audio signal recorded during the samereal period in measurement with the second system clock 220 is of aduration of 1001 seconds, then the audio signal can be processed in sucha way that it is re-scaled to a duration of 1000 seconds. For thedigitally recorded audio signal that can be carried out with a knownalgorithm for sample rate conversion. As a result an audio signal isthen available, the duration of which is nominally precisely 1000seconds and which also exactly matches the recorded video signal in thenumber of audio sample values ascertained in that way.

FIG. 1B shows a diagrammatic view of an external device 300 in whichprocessing of video and audio data can optionally be carried out.According to an aspect of the present invention the video signal of thevideo unit 100 and the audio signal of the microphone unit 200 can beprocessed in an external device 300. For that purpose the externaldevice can have a transmitter/receiver 310 and an audio signal samplingunit 320. The audio signal sampling unit can alternatively alsorepresent a part of the video signal 100.

According to an aspect of the present invention the video signal 100 canbe in the form of a smartphone.

FIG. 2 shows a diagrammatic view of system clocks of the video unit andthe audio unit according to the first embodiment. FIG. 2 shows a systemclock of the video unit 100 and a system clock of the audio unit 200 ona time axis. A first synchronization Sync1 is effected at the beginningof the audio and/or video recording and serves to detect the shiftbetween the system clock 120 of the video unit 100 and the system clock220 of the audio unit 200. The time difference between the first andsecond system clocks is constant when both system clocks are operated atthe same speed. As however by virtue of tolerances (tolerances in thequartzes, temperature fluctuations) the first and second system clocksare not running at the same speed in accordance with the invention asecond synchronization Sync2 is effected at the end of the audio and/orvideo recording. Because a first and a second synchronization Sync1,Sync2 is effected a difference in speed in the two system clocks can beascertained therefrom. If the speed difference is known it can besuitably compensated for example by re-sampling of the audio signals ofthe microphone 200.

To compensate for the differences in speed of the first and secondsystem clocks the audio signal of the microphone 200 can be sampledagain with an adapted sampling frequency. When that is done the videosignal then matches the new audio signal. The fresh sampling of theaudio signal can be ascertained by computation from the original audiosignal of the microphone 200.

According to an aspect of the present invention the audio signal and thevideo signal can be brought together in a device 300. That device 300can be for example a smartphone, tablet or the like which also has thevideo unit. As an alternative thereto it is possible to use an externaldevice.

FIG. 3 shows a further diagrammatic view of the system clocks of thevideo unit and the audio unit with system clocks which run at differentspeeds. FIG. 3 shows a video recording VR and an audio recording AR on atime axis. In that case a start of the recording occurs at S1 and a stopof the recording occurs at S2. The video recording VR begins at a timeS1, in relation to which the first system clock 120 specifies a time“17”. The video recording VR ends at a time S2, in relation to which thefirst system clock specifies a time “23”. At the time S1 the secondsystem clock 220 specifies a time “51” for the audio recording AR and atthe time S2 the second system clock 220 specifies a time “59”. As inthis example the second system clock 220 of the microphone unit 200 isrunning at a higher speed than the system clock of the video unit theaudio recording AR is nominally longer than the video recording. Thismeans that the two signals, with the same time base, are of differentlengths.

FIG. 4 shows a diagrammatic view of processing according to theinvention of the audio recording AR in accordance with FIG. 3 . FIG. 4shows in particular how correction of the sampling frequency of theaudio signal is effected. FIG. 4 shows the audio recording AR and thefreshly sampled audio recording AR1. In order to compensate for thedifferent speeds of the first and second system clocks the audio signalis again sampled by computation with an altered sampling frequency. Thatfreshly sampled audio signal is shown as the audio signal AR1 in FIG. 4. From the contained audio information relating to the representation ofa continuous audio signal the audio signal AR1 corresponds to theoriginally recorded audio recording AR but is re-scaled to a durationfrom a measurement time “17” to a measurement time “23” so that thesampling points of the originally recorded representation AR and therepresentation AR1 ascertained by computation differ from each other.

FIG. 5 shows a diagrammatic view of a corrected audio data file andvideo data file according to an aspect of the present invention. In FIG.5 the audio signal AR1 is now provided with the same system clock orsampling frequency as the video recording VR, that is to say those twosignals have the same time base. That provides that the audio recordingand the video recording AR1, VR are synchronous.

According to an aspect of the present invention instead of renewedsampling of the audio signal parts can be removed from the recordedaudio signal or synthetic portions can be added to reduce or increasethe length of the audio recording and to adapt the length of the audiorecording to the length of the video recording.

Even if a wireless connection is not available between the video unit100 and the microphone unit 200 at the beginning and/or end of a videorecording it is possible, from stored system clock alignment resultswhich were ascertained at different times, to approximately subsequentlycalculate in relation to each desired time value of the first systemclock 120, a time value that the second system clock 220 had at theassociation time. For that purpose it can be assumed that both systemclocks in themselves are running approximately at a constant speed.

FIG. 6 shows a diagrammatic view of the first system clock 120 and thesecond system clock 220 in relation to time t. A first alignment effecttook place at a first time T1, at which the first system clock 120 had atime value A1. It is known from that first alignment operation that thesecond system clock 220 had a time value B1 at that first time T1. Atanother second time T2 at which the first system clock 120 had a timevalue A2 a second alignment operation took place. It is known from thatsecond alignment operation that the second system clock 220 had a timevalue B2 at that second time T2.

If now for example a video recording began at a third time T3 at whichthe first system clock 120 had a time value A3 and which was stored as avideo time stamp with the video signal, at which however no alignment ofthe system clocks has occurred, the time value B3 that the second systemclock had at that third time T3 can be calculated according to thefollowing formula:B3=B1+(B2−B1)/(A2−A1)*(A3−A1)

FIG. 6 shows the situation where the third time T3 is between the twoalignment times T1 and T2. The formula however can also be applied ifthat is not the case. That can be advantageous for example if firstly nosystem clock alignment has occurred before the microphone unit 200 hasstored in its memory 250 an audio signal including associated audio timestamps and the video unit 100 has also stored in its memory 150 a videosignal including associated video time stamps, and wherein the videosignal has a video time stamp with a time value A3. If a first and asecond system clock alignment operation is now repeated, only after therecording of those two signals at different times T1 and T2, it isnonetheless possible by means of the specified formula to alsosubsequently approximately determine the time value B3 that the secondsystem clock 220 had at the time T3.

Basically therefore two system clock alignment results which wereascertained at two different times are sufficient to calculate at anytime value A3 of the first system clock 120 an associated time value B3that the second system clock 220 had at that time. To achieve a highlevel of accuracy alignment times which are in the proximity in respectof time of the respective times being considered should be used as faras possible for the calculation. It is also advantageous for accuracy ifthe two alignment times are not too close together.

The described calculation of associated time values of the first andsecond system clocks can be applied both to the beginning and also theend times of a video sequence, as well also for other values ofadditional time support locations.

System Clock Alignment by Way of Data Transfer Protocol:

As explained above alignment of the first system clock 120 with thesecond system clock 220 is based on simultaneously ascertaining the timevalue of the first system clock 120 and the second system clock 220. Itwill be noted however that many end devices like in particularsmartphones do not have available a wireless data transfer protocolwhich would ensure a predictable time shift in transmission.Transmission for example by way of WLAN, Bluetooth, DECT, LTE or WiFi asexplained above includes an unpredictable time shift so that timealignment by simply transferring a time value for example from the firstsystem clock 120 by way of such a transfer protocol to the microphoneunit could not afford the desired effect of a defined simultaneousascertainment of the time values of the two system clocks with adequateaccuracy.

Hereinafter therefore a method in accordance with an aspect of theinvention is set forth, in which alignment is implemented by particularuse of a data transfer protocol which is available on an end-user deviceand which is actually not intended to ensure a predictable time shift intransmission.

According to a second embodiment alignment is carried out by means of amethod which hereinafter is to be descriptively referred to as “pingpongsynchronization”. In that case a data packet is communicated from one ofthe devices to the other and then the second device communicates thedata packet together with a time value of its system clock back to thefirst device.

FIG. 7A shows a diagrammatic view of synchronization between a videounit and an audio unit according to a second embodiment of theinvention. FIG. 7A shows synchronization according to the invention on atime axis, wherein in the illustrated example the transmission of datafrom the video unit 100 to the microphone unit 200 and back involvesidentical transmission times. The video unit 100 firstly reads itscurrent time value t_(A,send) from the first system clock 120, puts thatvalue into interim storage and immediately sends a data packet by way ofan available data transfer protocol to the microphone unit 200. Thattransfer process PI (“ping”) starts alignment. The microphone unit 200receives the data packet and immediately responds by in turn reading itscurrent time value t_(B) out of the second system clock 220 andtransmitting that read-out time value with a second transfer process PO(“pong”) to the video unit 100. Upon receipt of the second transmissionPO the video unit 100 immediately again reads its current time valuet_(A,receive) out of the first system clock 120.

The video unit 100 therefore now knows the state t_(A,send) of the firstsystem clock 120 at the time of the beginning of the transfer PI and thestate t_(A,receive) of the first system clock 120 at the time of the endof the transfer PO. Then from those two time values read out of thefirst system clock 120 the video unit 100 can easily calculate byaveraging a time value t_(A,B) which is centrally between those two timevalues and which thus forms a first plausible estimate for the timevalue that the first system clock 120 had at the time at which the timevalue t_(B) was read out of the second system clock 220:t _(A,B) =t _(A,send)+(t _(A,receive) −t _(A,send))/2

FIG. 7A shows the optimum situation in which no uncontrollable delayshave occurred both in the first transfer PI and also in the secondtransfer PO. In addition FIG. 7A assumes an identical transfer time forthe first transfer PI and the second transfer PO. In the optimum caseshown in FIG. 7A the time value t_(A,B) calculated in that way evenexactly corresponds to the sought time value that the first system clock120 had at the time when the second system clock 220 output the timevalue t_(B).

In addition that time value t_(B) that the second system clock 220 hadat the time between the transfers PI and PO is available to the videounit 100 from the second transfer PO. Therefore an alignment result isavailable in the video unit 100 with t_(A,B) and t_(B), which resultrepresents the state of the two system clocks 120 and 220 at a commontime and which can be stored for later use in the memory 150. Thereforesimultaneous detection of the time values of the two system clocks iseffected so that the desired system clock alignment has succeeded.

Optionally the alignment results can also be transmitted to themicrophone unit 200 in a further data transfer and also stored there inthe memory 250. The described start of alignment can also be triggeredin a completely identical fashion by the microphone unit 200 so that themicrophone unit 200 and the video unit 100 then perform preciselyreversed roles.

As described in the opening part of this specification the availabledata transfer protocols—in particular caused by unknown waiting timesfor access to a transmission channel jointly used with otherdevices—involve unpredictable delay times. In the described pingpongsynchronization procedure both the first transfer operation PI and alsothe second transfer operation PO are thus undefinedly extended inlength. It is also not possible either for the respective transmittingdevice nor for the respective receiving device to ascertain the transfertime which has occurred in an individual transfer. The result of thiscan be that the calculated time value t_(A,B) differs from the actualtime value that the first system clock 120 had at the time when thesecond system clock 220 output the time value t_(B).

FIG. 7B shows a diagrammatic view of synchronization between a videounit 100 and an audio unit 200 for such a case. Here as in the optimumcase in FIG. 7A the first transfer PI was effected without a delay. Inthe second transfer PO however it was necessary to wait for use of thetransmission channel being used so that the second transfer PO lastedlonger. In calculation of the time value t_(A,B) from the time valuest_(A,send) and t_(A,receive) read out of the first system clock 120there is an error e_(A,B) which gives the deviation between thecalculated value t_(A,B) and the actual time value that the first systemclock 120 had at the time when the time value t_(B) was read out of thesecond system clock 220. As it is not known whether the delay occurredin the first transfer process PI or in the second transfer process PO orpossibly in both transfer processes the above-specified formula forcalculating the time value t_(A,B) however represents the best estimatewhich can be determined from a the single pingpong cycle.

FIG. 8 shows a diagrammatic view of synchronization measurement inaccordance with an aspect of the present invention. In the aim ofreducing the error e_(A,B) a plurality of transfer processes PI and POare here carried out in succession. FIG. 8 shows such a series of threepingpong cycles a, b and c between the video unit 100 and the microphoneunit 200 on a time axis.

In order now from the series of pingpong cycles to produce a timealignment with an error e_(A,B) which is as slight as possible thatpingpong cycle which had overall the shortest duration can be selectedfrom the series. In FIG. 8 that is the pingpong cycle a. The video unitcan recognize the shortest pingpong cycle on the basis of the timevalues t_(A,send) and t_(A,receive) stored for each pingpong cycle. Theshortest pingpong cycle evidently involves the least delays so that thisshortest cycle provides the best estimate for time alignment. If thetransmission channel is not overloaded there is a good chance that eventhe optimum case shown in FIG. 7A occurs, in which neither the firsttransfer process PI nor the second transfer process PO was delayed.Therefore, for time alignment only the shortest pingpong cycle, in theexample of FIG. 8 therefore only the pingpong cycle a, is used. The timevalue t_(A,B) is correspondingly only calculated from the time valuest_(A,send) and t_(A,receive) associated with the cycle a and associatedwith the time value t_(B) of the second system clock, that isascertained from the cycle a. Therefore a reduction in the error e_(A,B)is achieved by evaluation of a series of pingpong cycles to ascertain asingle time alignment effect.

If the transmission channel is so heavily loaded that the occurrence ofthe optimum situation shown in FIG. 7A is not to be expected for anindividual pingpong cycle then optionally a tolerance range for thealignment result can be determined by more precise analysis of a seriesof pingpong cycles. For that consideration firstly here a “clock offset”CO will be defined, which specifies the difference between a time valueof the second system clock 220 and a simultaneously ascertained timevalue of the first system clock 120. The clock offset CO therefore givesthe relationship between the first and second time bases. The actualvalue of CO is not known and is to be ascertained by measurement. As anestimate for CO for each pingpong cycle a shift O_(A,B) is specified,which gives the difference between the read-out time value t_(B) fromthe second system clock 220 and the time value t_(A,B), calculated inrelation thereto, of the first system clock 120, for a respectiveindividual pingpong cycle:O _(A,B) =t _(B) −t _(A,B)

In addition a minimum value t_(transmit,min) can be assumed to be knownfor the duration of an individual transfer process PI or PO respectively(as shown for the optimum case in FIG. 7A). From each individualpingpong cycle it is then possible to specify a maximum valuee_(A,B,max) for the error, that the calculated shift O_(A.B) can have inrelation to the value CO which is being sought:e _(A,B.max)=(t _(A,receive) −t _(A,send))/2−t _(transmit,min)

That greatest error occurs when only one of the transfer processes PI orPO has a delay, but the other is effected without any delay (see FIG.7B). The error e_(A,B) is therefore at a maximum as great as half themeasurement duration minus the shortest possible transmission timet_(transmit,min), wherein in this embodiment it is assumed that theshortest possible transmission time from the video unit 100 to themicrophone unit 200 (PI) corresponds to the shortest possibletransmission time from the audio unit 200 to the video unit 100 (PO). Ifa minimum value t_(transmit,min) for the duration of an individualtransmission process PI or PO is not known the value zero can be sethere.

For each individual pingpong cycle it is now possible on the basis ofthe respectively known possible error to specify an interval in whichthe actual value of a clock offset CO must be.

FIG. 9 shows a diagrammatic view of the shift, ascertained from asynchronization operation, of the system clocks of the video unit andthe audio unit. FIG. 9 shows on the time axis for the clock offset COthe intervals which occur in the FIG. 8 example from the three pingpongcycles a, b and c. A shift Oa was calculated from pingpong cycle a andthe maximum value of the error e_(a,max) calculated for cycle aspecifies respectively plotted negatively and positively the limits ofthe interval in which the actual value of a clock offset CO inaccordance with cycle a must lie. Intervals around Ob and Oc in whichthe actual value of a clock offset CO must lie are respectivelycorresponding plotted for cycles b and c.

There are different intervals for the various pingpong cycle, in whichthe actual value of the clock offset CO must lie. As the actual value ofCO must lie in each of the ascertained intervals the actual value of COcan only lie in a range in which all ascertained intervals overlap. InFIG. 9 that is characterised as a tolerance range from Omin throughOmax. As can be seen from FIG. 9 that tolerance range is normallysmaller than the intervals from the individual pingpong cycles. Uponcloser analysis it can be established that the difference from the lowerlimit Omin of the tolerance range to the correct value of CO correspondsto the duration of the shortest first transfer process PI from allconsidered cycles. This correspondingly means that the difference fromthe correct value of CO to the upper limit Omax of the tolerance rangecorresponds to the duration of the shortest second transfer process POfrom all considered cycles. If therefore an undelayed first transferprocess PI and an undelayed transfer process PO were at all contained inthe series of pingpong cycles the average value between Omin and Omaxprecisely specifies the correct value for CO.

From the value ascertained in that way for CO it is then possible, withone of the time values t_(B) read out of the second system clock 220during the measurements, to calculate an associated value t_(A,B) whichwith a high level of probability corresponds to the actual value thatthe first system clock 120 had, when the time value t_(B) was read outof the second system clock:t _(A,B) =t _(B) −CO

Those two time values t_(B) and t_(A,B) ascertained from a series ofpingpong cycles then form the result of time alignment which can then bestored in the memory 150 of the video unit and/or in the memory 250 ofthe microphone unit.

According to an aspect of the present invention with a Bluetoothconnection it is possible to achieve a measurement duration of 5 ms,whereby a maximum possible error in synchronization is at 0.5 ms.

According to the invention a block-based transmission is effected by wayof a wireless channel, the blocks being of different latencies.According to the invention a synchronization signal is to be producedthe video signal and the wirelessly received audio signal. Themicrophone unit 200 can store time stamps in the audio signal. The videounit 100 can store time stamps in the video signal.

Typical Situation of Use of the Overall System:

A typical situation of use of the overall system is describedhereinafter: before the beginning of an event a bidirectional wirelessconnection is made for the purposes of the transmission of the audiodata and alignment of the system clocks by way of thetransmitter/receiver 110 in the video unit 100 (in particular asmartphone) and the transmitter/receiver 210 in the microphone unit 200.That involves using an individualized data transfer protocol which isavailable on the video unit 100 and in which there is provided a datastream for a respective given end-user device and which is transmittedsubdivided into individual data packets to the receiving end device. Thedata transfer protocol is characterized in that coordinated use of atransmission channel involved is provided by a plurality of transmittingand receiving devices and the data transfer protocol has measures fortime coordination of the access of various devices to the transmissionchannel. The result of this is that variable waiting times can occur foraccess to the transmission channel so that the time implementation oftransmission of a respective data packet cannot be accurately predicted.As examples mention will be made here of WLAN, Bluetooth, LTE and Wi-Fi.

As soon as the wireless connection exists a first system clock alignmentis carried out in accordance with the foregoing description and theresult of alignment is stored in the memory 150 of the video unit and/orin the memory 250 of the microphone unit. Alignments of the systemclocks are again planned by way of the wireless connection regularly,for example at spacings of a second. A result of the alignment procedurehowever is stored only if the wireless connection respectively permitsalignment. The bidirectional wireless connection is classified asconnected in the further proceeding from both sides during the event,even if no data transfer occurs over a prolonged period of time.

In parallel with the constitution of the bidirectional wirelessconnection the microphone unit 200 begins therewith continuously, thatis to say without interruptions, to record an audio signal with theassociated audio time stamps and to store the result in its memory 250.The microphone unit 200 is then taken during the event to the locationat which the microphone signal is to be detected, that is to say forexample on the stage, or it is fastened as a clip-on microphone to aperformer or is held in the hand by the performer. During the event thevideo unit 100 is taken to a location from which a video recording is tobe made. This can be a position in the auditorium, from which the userof a smartphone would like to make video recordings. By virtue of thespatial distance between the microphone unit 200 and the video unit 100and possibly numerous competing access attempts for accessing thetransmission channel used ongoing existence of the wireless connectioncan possibly not be reliably guaranteed during the event.

Typically only individual sections of an event are recorded with thevideo unit 100. If the user of the video unit 100 now starts to record avideo sequence it would normally be necessary to transmit that startinformation to the microphone unit 200 as far as possible at the sametime by way of the wireless connection so that it begins at the sametime as the recording of an associated audio sequence and thetransmission thereof. In the typical use described herein of a systemaccording to the invention however, having regard to the background ofan unreliable wireless connection, it is possible to dispense with thatactivation of the microphone unit as the audio recording is in factcarried out continuously together with storage in the memory 250.Instead the video unit transmits to the microphone unit by way of thewireless connection time information which specifies the time of thebeginning of the video sequence—based on the first system clock 120. Inthat case the time information can be transmitted in the form of thefirst time base. The microphone unit 200 then calculates on the basis ofthe previous time alignment results the associated time value of thesecond system clock 220. Alternatively conversion to the second timebase can already be effected on the basis of previous time alignmentresults prior to transmission in the video unit 100 so that thetransmitted time information already specifies the associated time valueof the second system clock 220. If the described time information shouldonly arrive at the microphone unit with a delay due to a delayedtransmission then nonetheless as described with reference to FIG. 6 thecorresponding time can be found in the stored audio signal on the basisof the stored audio time stamps and the existing alignment informationfor the two system clocks. The specified transmission of the videobeginning time information serves at the same time as a request to themicrophone unit 200 to transmit a part from the stored audio signal tothe video unit 100. The microphone 200 in response to that requesttransmits a part of the stored audio signal which begins at said time tothe video unit 100 by way of the wireless connection. When the user endsthe video sequence in the same way time information specifying the timeof the end of the video sequence can be transmitted from the video unitto the microphone unit and thus the end of the part to be transmitted ofthe stored audio signal can be established. The stored audio timesignals are also transmitted with the audio signal.

As soon as the video unit has received the audio/time stamp signal for agiven video sequence the video unit assigns the received audio signalcorrectly in respect of time to the video signal on the basis of theaudio and the video time stamps and on the basis of stored system clockalignment results, possibly carries out the above-described timere-scaling of the audio signal to align differences in speed of thesystem clocks and stores the combination of the video signal and theaudio signal processed in that way in the memory 150 of the video unit.

If the wireless connection is not available at the time of the videorecording then the video unit (without user intervention) repeats therequests for missing parts of the audio/time stamp signal as soon asdata can be transmitted again by way of the connection and thensubsequently carries out the described procedure for bringing audio andvideo sequences together correctly in respect of time. At the latest ifthe user of the video unit takes possession of the microphone unit 200after the end of the event and in so doing the microphone unit 200 andthe video unit 100 are in immediate proximity with each other a datatransfer can take place and the missing audio data relating to the videosequences produced during the event can be transmitted to the videounit. Without further intervention on the part of the user he willtherefore then find on his video unit audio-video sequences in which theaudio signals recorded with the microphone unit 200 are stored incorrect time relationship with the video sequences produced during theevent.

Alternative Possible Uses:

The described method of recording and synchronizing audio and videosignals can be used in the same manner for recording and synchronizingother signal configurations which are detected in relation to time. Forexample the time alignment of two system clocks as described withreference to FIGS. 7 through 9 can also be used for the synchronizationof two video signals. That can be useful for example if video recordingsare produced from the same event by two or more smartphones. Upon laterprocessing of the recorded video signals the user would like to cut avideo together from the recordings made by the two cameras, in whichcase he can switch at the correct time from the recording of one camerato a recording of the other camera. In this case also there is theproblem that wireless alignment of the system clocks of the twosmartphones may suffer from an unknown delay so that the above-describedmethod of equalizing the two system clocks can produce an improvedalignment result.

The method in still broader terms can also be applied to completelydifferent signal configurations in relation to time like for example thespeed of a vehicle or a temperature pattern in a chemical reaction. Themicrophone 230 or the camera 130 can then be generally viewed as asignal detection sensor and the detected signal is stored in arespective recording device together with time stamps.

The method according to the invention can be used whenever signalconfigurations on two separate recording devices are respectivelydetected with a dedicated time base and stored, which are later to bebrought together in the correct time relationship, and wherein awireless communication between the two recording devices may suffer froman unknown delay.

The invention claimed is:
 1. A microphone unit comprising a microphonefor recording audio signals, a transmitter/receiver for wirelessbidirectional communication with a video unit which has a first systemclock with a first time base, a second system clock with a second timebase, and a memory for digital storage of an audio signal recorded withthe microphone and time synchronization information; wherein themicrophone unit is adapted to repeatedly store in the memory jointlywith an audio signal recorded by the microphone audio time stamps whichspecify the respective times of recording of the audio signal measuredwith the second system clock; wherein the transmitter/receiver isadapted to communicate with the video unit by way of a data transferprotocol which is provided for a coordinated use of a used transmissionchannel by a plurality of transmitting and receiving devices and whichhas measures for time coordination of the access of different devices tothe transmission channel so that variable waiting times can occur foraccess to the transmission channel; wherein the microphone unit isadapted by way of the transmitter/receiver to carry out an alignmentbetween the first system clock and the second system clock, wherein thealignment result includes a time value of the first system clock and anassociated time value of the second system clock; and wherein thealignment is effected by the video unit or the microphone unit as theinitiating unit firstly reading out a current transmission time value(t_(a,send)) from its system clock and temporarily storing it andimmediately initiating a first transfer process (PI) to the other of thetwo units, and wherein the respective other unit after conclusion of thefirst transfer process (PI) immediately reads out a currentsynchronization time value (t_(B)) from its system clock and transmitssaid synchronization time value (t_(B)) in a second transfer process(PO) to the initiating unit, and wherein the initiating unit uponreception of the second transfer (PO) immediately reads a currentreception time value (t_(A,receive)) from its system clock and from thetransmission time value (t_(A,send)) and the reception time value(T_(A,receive)) calculates a time value belonging to the synchronizationtime value (t_(B)).
 2. An audio/video recording and synchronizationsystem comprising the microphone unit as set forth in claim 1 and avideo unit, wherein the video unit comprises a camera unit for recordingvideo signals, a transmitter/receiver for wireless bidirectionalcommunication with the microphone unit, a first system clock with afirst time base, and a memory for digital storage of a video signalrecorded with the camera unit and time synchronization information;wherein the video unit is adapted to repeatedly store in the memorytogether with a video signal recorded by the camera unit video timestamps which specify the respective times of recording of the videosignal measured with the first system clock; and wherein the video unitis adapted by way of the transmitter/receiver to carry out an alignmentbetween the first system clock and the second system clock.
 3. A methodof recording and synchronizing audio and video signals comprising thesteps: recording a video signal by means of a video unit and storing thevideo signal together with video time stamps which specify therespective times of recording of the video signal measured with a firstsystem clock; recording an audio signal by means of a microphone unitand storing the audio signal together with audio time stamps whichspecify the respective times of recording of the audio signal measuredwith a second system clock; carrying out an alignment between the firstsystem clock and the second system clock, wherein the alignment resultcontains a time value of the first system clock and an associated timevalue of the second system clock; and bringing together the audio signaland the video signal, wherein time synchronization is effected on thebasis of the video time stamps, the audio time stamps and the alignmentresult; wherein the alignment between the first system clock and thesecond system clock is effected by way of a bidirectional wirelessconnection between the video unit and the microphone unit; wherein adata transfer protocol is used, which is provided for a coordinated useof a used transmission channel by a plurality of transmitting andreceiving devices and which has measures for time coordination of theaccess of different devices to the transmission channel so that variablewaiting times can occur for access to the transmission channel; andwherein the alignment is effected by the video unit or the microphoneunit as the initiating unit firstly reading out a current transmissiontime value (t_(a,send)) from its system clock and temporarily storing itand immediately initiating a first transfer process (PI) to the other ofthe two units, and wherein the respective other unit after conclusion ofthe first transfer process (PI) immediately reads out a currentsynchronization time value (t_(B)) from its system clock and transferssaid synchronization time value (t_(B)) in a second transfer process(PO) to the initiating unit, and wherein the initiating unit uponreception of the second transfer (PO) immediately reads a currentreception time value (t_(A,receive)) from its system clock and from thetransmission time value (t_(A,send)) and the reception time value(T_(A,receive)) calculates a time value belonging to the synchronizationtime value (t_(B)).
 4. A method of recording and synchronizing aplurality of signal configurations in relation to time, comprising thesteps: recording a first signal configuration by means of a firstrecording device and storing the first signal configuration togetherwith first time stamps which specify the respective times of recordingof the first signal measured with a first system clock contained in thefirst recording device; recording a second signal configuration by meansof a second recording device and storing the second signal configurationtogether with second time stamps which specify the respective times ofrecording of the second signal measured with a second system clockcontained in the second recording device; carrying out an alignmentbetween the first system clock and the second system clock, wherein thealignment result includes a time value of the first system clock and anassociated time value of the second system clock; wherein the alignmentbetween the first system clock and the second system clock is effectedby way of a bidirectional wireless connection between the firstrecording device and the second recording device; wherein a datatransfer protocol is used, which is provided for a coordinated use of aused transmission channel by a plurality of transmitting and receivingdevices and which has measures for time coordination of the access ofdifferent devices to the transmission channel so that variable waitingtimes can occur for access to the transmission channel; and wherein thealignment is effected by the first recording device or the secondrecording device as the initiating unit firstly reading out a currenttransmission time value (t_(a,send)) from its system clock andtemporarily storing it and immediately initiating a first transferprocess (PI) to the other of the two units, and wherein the secondrecording device after conclusion of the first transfer process (PI)immediately reads out a current synchronization time value (t_(B)) fromthe second system clock and transfers said synchronization time value(t_(B)) in a second transfer process (PO) to the first recording device,and wherein the first recording device upon reception of the secondtransfer (PO) immediately reads out a current reception time value(t_(A,receive)) from the first system clock and from the transmissiontime value (t_(A,send)) and the reception time value (T_(A,receive))calculates a time value belonging to the synchronization time value(t_(B)).
 5. The method of recording and synchronizing a plurality ofsignal configurations in relation to time as set forth in claim 4,additionally comprising the step: bringing together the first signalconfiguration and the second signal configuration, wherein timesynchronization is effected on the basis of the first time stamps, thesecond time stamps and the alignment result.
 6. A recording device forrecording a signal configuration in relation to time, comprising asignal recording sensor for recording a signal, a first system clockwith a first time base, a memory for digital storage of a signalconfiguration recorded with the signal recording sensor in relation totime and time synchronization information, and a transmitter/receiverfor wireless bidirectional communication with a second recording devicewhich has a second system clock with a second time base; wherein therecording device is adapted to repeatedly store in the memory jointlywith a signal configuration recorded by the signal recording sensor timestamps which specify the respective times of recording of the signalmeasured with the first system clock; wherein the transmitter/receiveris adapted to communicate with the second recording device by way of adata transfer protocol which is provided for a coordinated use of a usedtransmission channel by a plurality of transmitting and receivingdevices and which has measures for time coordination of the access ofdifferent devices to the transmission channel so that variable waitingtimes can occur for access to the transmission channel; wherein therecording device is adapted by way of the transmitter/receiver to carryout an alignment between the first system clock and the second systemclock, wherein the alignment result includes a time value of the firstsystem clock and an associated time value of the second system clock;and wherein the alignment is effected by the recording device or thesecond recording device as the initiating unit firstly reading out acurrent transmission time value (t_(a,send)) from its system clock andtemporarily storing it and immediately initiating a first transferprocess (PI) to the other of the two recording devices, and wherein therespective other recording device after conclusion of the first transferprocess (PI) immediately reads out a current synchronization time value(t_(B)) from its system clock and transmits said synchronization timevalue (t_(B)) in a second transfer process (PO) to the initiatingrecording device, and wherein the initiating recording device uponreception of the second transfer (PO) immediately reads out a currentreception time value (t_(A,receive)) from its system clock and from thetransmission time value (t_(A,send)) and the reception time value(T_(A,receive)) calculates a time value belonging to the synchronizationtime value (t_(B)).